Papers with speech-to-speech translation
i-Code Studio: A Configurable and Composable Framework for Integrative AI (2024.emnlp-demo)
Copied to clipboard
Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang
| Challenge: | Existing frameworks for Integrative AI lack flexibility and composability to handle multimodal tasks. |
| Approach: | They propose a configurable framework for Integrative AI that orchestrates multiple pre-trained models to conduct complex multimodal tasks. |
| Outcome: | The proposed framework achieves impressive results on zero-shot multimodal tasks . it can communicate and personalize for users, and it can be used in a multimodal agent . |
EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | EmphAssess evaluates speech-to-speech models' ability to encode and reproduce prosodic emphasis across a change of speaker and language. |
| Approach: | They propose a prosodic benchmark to evaluate the ability of speech-to-speech models to encode and reproduce prosodic emphasis. |
| Outcome: | The proposed model can encode and reproduce prosodic emphasis across speech inputs and outputs . EmphaClass classifies emphasis at the frame or word level . |
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing translation pipelines require additional cascade components to achieve speech-to-speech translation. |
| Approach: | They propose a non-autoregressive generation framework for simultaneous speech translation . it integrates both text-to-text and speech-tospeech tasks into a unified framework . |
| Outcome: | The proposed framework outperforms state-of-the-art models in speech-to-text and speech- to-speech tasks. |
LibriS2S: A German-English Speech-to-Speech Translation Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent advances in speech-to-text translation have led to significant improvements, but the availability of appropriate training data is limiting. |
| Approach: | They propose a new text-to-speech and speech-tospech translation model that directly learns to generate the speech signal based on the pronunciation of the source language. |
| Outcome: | The proposed model learns to generate speech signal based on pronunciation of source language. |
Aligning Speech Segments Beyond Pure Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing speech-to-speech parallel data is scarce and expensive to create from scratch. |
| Approach: | They propose an algorithm which automatically aligns pairs of speech segments aligned in meaning and expressivity. |
| Outcome: | The proposed algorithm outperforms semantic-focused approaches on content translation quality. |
Direct Speech-to-Speech Translation With Discrete Units (2022.acl-long)
Copied to clipboard
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, Wei-Ning Hsu
| Challenge: | Existing direct speech-to-speech translation models rely on text generation as an intermediate step. |
| Approach: | They propose a direct speech-to-speech translation model that translates speech from one language to another without relying on intermediate text generation. |
| Outcome: | The proposed model produces 6.7 BLEUs in the Fisher Spanish-English dataset when trained without any text transcripts and with text supervision. |
Speech-to-Speech Translation for a Real-world Unwritten Language (2023.findings-acl)
Copied to clipboard
Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, Ann Lee
| Challenge: | a new study examines speech-to-speech translation (S2ST) that translates speech from one language into another . the research area for unwritten languages remains a research area with little exploration due to the lack of training data. |
| Approach: | They propose a system that translates speech from one language into another . they use Taiwanese Hokkien as an example of an unwritten language . |
| Outcome: | The proposed system can be used to train models in languages without standard writing systems. |
Textless Speech-to-Speech Translation With Limited Parallel Data (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing speech-to-speech translation models either leverage text as an intermediate step or require hundreds of hours of parallel speech data. |
| Approach: | They propose a framework for training textless S2ST models that require dozens of hours of parallel speech data. |
| Outcome: | The proposed model achieves reasonable performance on three domains with single-speaker synthesized speech. |
S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | S2ST-Omni integrates a speech-to-text frontend with a modular, plug-and-play text-tospeech backend. |
| Approach: | They propose a compositional S2ST framework that integrates a speech-to-text frontend with a modular, plug-and-play text-tospeech backend. |
| Outcome: | The proposed framework outperforms existing frameworks in translation and synthesis . it integrates a speech-to-text translation frontend with a plug-and-play text-tospeech backend . |